Remote sensing for detection and ranging of objects

ABSTRACT

Aspects of the present disclosure involve example systems and methods for remote sensing of objects. In one example, a system includes a light source, a camera, a light radar (lidar) system, a region of interest identification circuit, and a range refining circuit. The region of interest identification circuit identifies a region of interest corresponding to an object and a first range of distance to the region of interest using the camera and the light source. The range refining circuit probes the region of interest to refine the first range to a second range of distance to the region of interest using the lidar system, the second range having a lower uncertainty than the first range.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to and claims priority under 35 U.S.C. §119(e) to U.S. Patent Application No. 62/233,211, filed Sep. 25, 2015,entitled “Remote Sensing for Detection and Ranging of Objects” and U.S.Patent Application No. 62/233,224, filed Sep. 25, 2015, entitled “RemoteSensing for Detection and Ranging of Objects,” both of which are herebyincorporated by reference in their entirety.

TECHNICAL FIELD

This disclosure relates generally to remote sensing systems, and morespecifically to remote sensing for the detection and ranging of objects.

BACKGROUND

Remote sensing, in which data regarding an object is acquired usingsensing devices not in physical contact with the object, has beenapplied in many different contexts, such as, for example, satelliteimaging of planetary surfaces, geological imaging of subsurfacefeatures, weather forecasting, and medical imaging of the human anatomy.Remote sensing may thus be accomplished using a variety of technologies,depending on the object to be sensed, the type of data to be acquired,the environment in which the object is located, and other factors.

One remote sensing application of more recent interest is terrestrialvehicle navigation. While automobiles have employed different types ofremote sensing systems to detect obstacles and the like for years,sensing systems capable of facilitating more complicated functionality,such as autonomous vehicle control, remain elusive.

SUMMARY

According to one embodiment, a system for sensing objects includes alight source, a camera, a light radar (lidar) system, a region ofinterest identification circuit to identify a region of interestcorresponding to an object and a first range of distance to the regionof interest using the camera and the light source, and a range refiningcircuit to probe the region of interest to refine the first range to asecond range of distance to the region of interest using the lidarsystem, the second range having a lower uncertainty than the firstrange.

According to another embodiment, a method for sensing objects includesidentifying, by a processor, a region of interest corresponding to anobject and a first range of distance to the region of interest using acamera and a light source, and probing, by the processor, the region ofinterest to refine the first range to a second range of distance to theregion of interest using a lidar system, the second range having a loweruncertainty than the first range.

In an additional embodiment, a non-transitory computer-readable mediumstores instructions that, when executed by a processor, cause theprocessor to perform operations for sensing objects, the operationsincluding identifying a region of interest corresponding to an objectand a first range of distance to the region of interest using a cameraand a light source, and probing the region of interest to refine thefirst range to a second range of distance to the region of interestusing a lidar system, the second range having a lower uncertainty thanthe first range.

These and other aspects, features, and benefits of the presentdisclosure will become apparent from the following detailed writtendescription of the preferred embodiments and aspects taken inconjunction with the following drawings, although variations andmodifications thereto may be effected without departing from the spiritand scope of the novel concepts of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example sensing system including aninfrared camera operating in conjunction with a controllable lightsource.

FIG. 2A is a timing diagram of a light pulse generated by a light sourceand an exposure window for the infrared camera of FIG. 1 for generalscene illumination.

FIG. 2B is a timing diagram of a light pulse generated by a light sourceand multiple overlapping exposure windows for the infrared camera ofFIG. 1 for close-range detection of objects.

FIG. 2C is a timing diagram of a light pulse generated by a light sourceand an exposure window for the infrared camera of FIG. 1 forlonger-range detection of objects.

FIG. 2D is a timing diagram of a light pulse generated by a light sourceand multiple distinct exposure windows for the infrared camera of FIG. 1for relatively fine resolution ranging of objects.

FIG. 2E is a timing diagram of a light pulse generated by a light sourceand multiple overlapping exposure windows for the infrared camera ofFIG. 1 for fine resolution ranging of objects.

FIG. 3 is a timing diagram of light pulses of one sensing systemcompared to an exposure window for the infrared camera of FIG. 1 of adifferent sensing system in which pulse timing diversity is employed tomitigate intersystem interference.

FIG. 4 is a graph of multiple wavelength channels for the light sourceof FIG. 1 to facilitate wavelength diversity to mitigate intersysteminterference.

FIG. 5A is a block diagram of an example multiple-camera system tofacilitate multiple fields of view.

FIG. 5B is a block diagram of an example multiple-camera system tofacilitate multiple depth zones for the same field of view.

FIG. 6 is a flow diagram of an example method of using an infraredcamera for fine resolution ranging.

FIG. 7 is a block diagram of an example sensing system including aninfrared camera operating in conjunction with a controllable lightsource, and including a light radar (lidar) system.

FIG. 8A is a block diagram of an example lidar system using a rotatablemirror.

FIG. 8B is a block diagram of an example lidar system using atranslatable lens.

FIG. 9 is a flow diagram of an example method of employing an infraredcamera and a lidar system for fine range resolution.

FIG. 10 is a block diagram of an example vehicle autonomy system inwhich infrared cameras, lidar systems, and other components may beemployed.

FIG. 11 is a flow diagram of an example method of operating a vehicleautonomy system.

FIG. 12 is a functional block diagram of an electronic device includingoperational units arranged to perform various operations of thepresently disclosed technology.

FIG. 13 is an example computing system that may implement varioussystems and methods of the presently disclosed technology.

DETAILED DESCRIPTION

Aspects of the present disclosure involve systems and methods for remotesensing of objects. In at least some embodiments, remote sensing isperformed using a camera (e.g., an infrared camera) and associated lightsource, wherein an exposure window for the camera is timed relative topulsing of the light source to enhance the ranging information yielded.In some examples, the cameras may be employed in conjunction with lightradar (lidar) systems to identify regions of interest that, when coupledwith the enhanced ranging information (e.g., range-gated information),may be probed using the lidar systems to further improve the ranginginformation.

The various embodiments described herein may be employed in anautonomous vehicle, possibly in connection with other sensing devices,to facilitate control of acceleration, braking, steering, navigation,and other functions of the vehicle in various challenging environmentalconditions during the day or at night.

FIG. 1 is a block diagram of an example sensing system 100 that includesan imaging device, such as an infrared camera 104, operating inconjunction with a controllable light source 102 for sensing an object101. In other examples, cameras and associated light sources employingwavelength ranges other than those in the infrared range may besimilarly employed in the manner described herein. In one example, acontrol circuit 110 may control both the infrared camera 104 and thelight source 102 to obtain ranging or distance information of the object101. In the embodiment of FIG. 1, the control circuit 110 may include alight source timing circuit 112, an exposure window timing circuit 114,and a range determination circuit 116. Each of the control circuit 110,the light source timing circuit 112, the exposure window timing circuit114, and the range determination circuit 116 may be implemented ashardware and/or software modules. Other components or devices notexplicitly depicted in the sensing system 100 may also be included inother examples.

The light source 102, in one embodiment, may be an infrared lightsource. More specifically, the light source may be a near-infrared (NIR)light source, such as, for example, a vertical-cavity surface-emittinglaser (VCSEL) array or cluster, although other types of light sourcesmay be utilized in other embodiments. Each of multiple such lasersources may be employed, each of which may be limited in output power(e.g., 2-4 watts (W) per cluster) and spaced greater than some minimumdistance (e.g., 250 millimeters (mm)) apart to limit the amount ofpossible laser power being captured by the human eye. Such a lightsource 102 may produce light having a wavelength in the range of 800 to900 nanometers (nm), although other wavelengths may be used in otherembodiments. To operate the light source 102, the light source timingcircuit 112 may generate signals to pulse the light source 102 accordingto a frequency and/or duty cycle, and may alter the timing of the pulsesaccording to a condition, as described in the various examples presentedbelow.

The infrared camera 104 of the sensing system 100 may capture images ofthe object within a field of view (FOV) 120 of the infrared camera 104.In some examples, the infrared camera 104 may be a near-infrared (NIR)camera. More specifically, the infrared camera 104 may be a high dynamicrange NIR camera providing an array (e.g., a 2K×2K array) of imagingelements to provide significant spatial or lateral resolution (e.g.,within an x, y plane facing the infrared camera 104). To operate theinfrared camera 104, the exposure window timing circuit 114 may generatesignals to open and close an exposure window for the infrared camera 104to capture infrared images illuminated at least in part by the lightsource 102. Examples of such timing signals are discussed more fullyhereafter.

The range determination circuit 116 may receive the images generated bythe infrared camera 104 and determine a range of distance from theinfrared camera 104 to each of the objects 101. For example, the rangedetermination circuit 116 may generate both two-dimensional (2D) imagesas well as three-dimensional (3D) range images providing the rangeinformation for the objects. In at least some examples, the determinedrange (e.g., in a z direction orthogonal to an x, y plane) for aparticular object 101 may be associated with a specific area of the FOV120 of the infrared camera 104 in which the object 101 appears. Asdiscussed in greater detail below, each of these areas may be considereda region of interest (ROI) to be probed in greater detail by otherdevices, such as, for example, a lidar system. More generally, the datagenerated by the range determination circuit 116 may then cue a lidarsystem to positions of objects and possibly other ROIs for furtherinvestigation, thus yielding images or corresponding information havingincreased spatial, ranging, and temporal resolution.

The control circuit 110, as well as other circuits described herein, maybe implemented using dedicated digital and/or analog electroniccircuitry. In some examples, the control circuit 110 may includemicrocontrollers, microprocessors, and/or digital signal processors(DSPs) configured to execute instructions associated with softwaremodules stored in a memory device or system to perform the variousoperations described herein.

While the control circuit 110 is depicted in FIG. 1 as employingseparate circuits 112, 114, and 116, such circuits may be combined atleast partially. Moreover, the control circuit 110 may be combined withother control circuits described hereafter. Additionally, the controlcircuits disclosed herein may be apportioned or segmented in other waysnot specifically depicted herein while retaining their functionality,and communication may occur between the various control circuits inorder to perform the functions discussed herein.

FIGS. 2A through 2E are timing diagrams representing different timingrelationships between pulses of the light source 102 and the opening andclosing of exposure windows or gates for the infrared camera 104 forvarying embodiments. Each of the timing diagrams within a particularfigure, as well as across different figures, is not drawn to scale tohighlight various aspects of the pulse and window timing in each case.

FIG. 2A is a timing diagram of a recurring light pulse 202A generated bythe light source 102 under control of the light source timing circuit112 and an exposure window 204A or gate for the infrared camera 104under the control of the exposure window timing circuit 114 for generalscene illumination to yield images during times when ranging informationis not to be gathered, such as for capturing images of all objects andthe surrounding environment with the FOV 120 of the infrared camera 104.In this operational mode, the light source 102 may be activated (“on”)periodically for some relatively long period of time (e.g., 5milliseconds (ms)) to illuminate all objects 101 within some range ofthe infrared camera 104 (which may include the maximum range of theinfrared camera 104), and the exposure window 204A may be open duringthe times the light source 102 is activated. As a result, all objects101 within the FOV 120 may reflect light back to the infrared camera 104while the exposure window 204A is open. Such general scene illuminationmay be valuable for initially identifying or imaging the objects 101 andthe surrounding environment at night, as well as during the day as asort of “fill-in flash” mode, but would provide little-to-no ranginginformation for the objects 101. Additionally, pulsing the lights source102 in such a manner, as opposed to leaving the light source 102activated continuously, may result in a significant savings of systempower.

FIG. 2B is a timing diagram of a recurring light pulse 202B and multipleoverlapping exposure windows 204B for the infrared camera 104 forclose-range detection and ranging of objects 101 and 201. In thisexample, each light pulse 202B is on for some limited period of time(e.g., 200 nanoseconds (ns), associated with a 30 meter (m) range ofdistance), and a first exposure window 204B is open through the sametime period (TFULL) that the light pulse 202B is active, resulting in anassociated outer range 210B within 30 m from the infrared camera 104. Asecond exposure window 205B is opened at the same time as the firstexposure window 204B, but is then closed halfway through the time thelight pulse 202B is on (THALF) and the first exposure window 204B isopen, thus being associated with an inner range 211B within 15 m of theinfrared camera 104. Images of objects beyond these ranges 210B and 211B(e.g., in far range 214B), will not be captured by the infrared camera104 using the light pulse 202B and the exposure windows 204B and 205B.Accordingly, in this particular example, a returned light pulse 221Bfrom object 101 of similar duration to the light pulse 202B, as shown inFIG. 2B, will be nearly fully captured during both exposure window 204Band 205B openings due to the close proximity of object 101 to theinfrared camera 104, while a returned light pulse 222B from object 201will be partially captured during both the first exposure window 204Bopening and the second exposure window 205B opening to varying degreesbased on the position of object 201 within the ranges 210B and 211B, butmore distant from the infrared camera 104 than object 101.

Given the circumstances of FIG. 2B, the distance of the objects 101 and201 may be determined if a weighted image difference based on the twoexposure windows 204B and 205B is calculated. Since the light musttravel from the light source 102 to each of the objects 101 and 201 andback again, twice the distance from the infrared camera 104 to theobjects 101 and 201 (2(Δd)) is equal to the time taken for the light totravel that distance (Δt) times the speed of light (c). Stateddifferently:

${\Delta\; d} = {\frac{c}{2}( {\Delta\; t} )}$

Thus, for each of the objects 101 and 201 to remain within the innerrange 211B:

$0 \leq {\Delta\; d} \leq {\frac{c}{2}( T_{HALF} )}$

Presuming the rate at which the voltage or other response of an imagingelement of the infrared camera 104 rises while light of a particularintensity is being captured (e.g., while the exposure window 204B or205B is open), the range determination circuit 116 may calculate thetime Δt using the voltage associated with the first exposure window 204B(VFULL) and the voltage corresponding with the second exposure window205B (VHALF):Δt=(V _(FULL) T _(HALF) −V _(HALF) T _(FULL))/(V _(FULL) −V _(HALF))

The range determination circuit 116 may then calculate the distance Δdfrom the infrared camera 104 to the object 101 or 201 using therelationship described above. If, instead, an object lies outside theinner range 211B but still within the outer range 210B, the rangedetermination circuit 116 may be able to determine that the object liessomewhere inside the outer range 210B, but outside the inner range 211B.

FIG. 2C is a timing diagram of a recurring light pulse 202C generated bythe light source 102 and an exposure window 204C for the infrared camera104 for longer-range detection of objects. In this example, a short(e.g., 100 nsec) light pulse 202C corresponding to a 15 m pulse extentis generated, resulting in a returned light pulse 221C of similarduration for an object 101. As shown in FIG. 2C, each light pulse 202Cis followed by a delayed opening of the exposure window 204C of a timeperiod associated with a photon collection zone 210C in which the object101 is located within the FOV 120 of the infrared camera 104. Morespecifically, the opening of the exposure window 204C corresponds withthe edge of the photon collection zone 210C adjacent to a near-rangeblanking region 212C, while the closing of the exposure window 204Ccorresponds with the edge of the photon collection zone 210C adjacent toa far-range blanking region 214C. Consequently, use of the recurringlight pulse 202C and associated exposure window 204C results in arange-gating system in which returned pulses 221C of objects 101residing within the photon collection zone 210C are captured at theinfrared camera 104. In one example, each exposure window 204C may beopen for 500 nsec, resulting in a photon collection zone 210C of 75 m indepth. Further, a delay between the beginning of each light pulse 202Cand the beginning of its associated opening of the exposure window 204Cof 500 nsec may result in a near-range blanking region 212C of 75 mdeep. Other depths for both the near-blanking region 212C and the photoncollection zone 210C may be facilitated based on the delay and width,respectively, of the exposure window 204C opening in other embodiments.Further, while the light pulse 202C of FIG. 2C is of significantlyshorter duration than the exposure window 204C, the opposite may be truein other embodiments.

Further, in at least some examples, the width or duration, along withthe intensity, of each light pulse 202C may be controlled such that thelight pulse 202C is of sufficient strength and length to allow detectionof the object 101 within the photon collection zone 210C at the infraredcamera 104 while being short enough to allow detection of the object 101within the photon collection zone 210C within some desired level ofprecision.

In one example, a voltage resulting from the photons collected at theinfrared camera 104 during a single open exposure window 204C may beread from each pixel of the infrared camera 104 to determine thepresence of the object 101 within the photon collection zone 210C. Inother embodiments, a voltage resulting from photons collected duringmultiple such exposure windows 204C, each after a corresponding lightpulse 202C, may be read to determine the presence of an object 101within the photon collection zone 210C. The use of multiple exposurewindows 204C in such a manner may facilitate the use of a lower powerlight source 102 (e.g., a laser) than what may otherwise be possible. Toimplement such embodiments, light captured during the multiple exposurewindows 204C may be integrated during photon collection on an imagerintegrated circuit (IC) of the infrared camera 104 using, for example,quantum well infrared photodetectors (QWIPs) by integrating the chargecollected at quantum wells via a floating diffusion node. In otherexamples, multiple-window integration of the resulting voltages mayoccur in computing hardware after the photon collection phase.

FIG. 2D is a timing diagram of a recurring light pulse 202D generated bythe light source 102 and multiple distinct exposure windows 204D and205D for the infrared camera 104 for relatively fine resolution rangingof objects. In this example, the openings of the two exposure windows204D and 205D are adjacent to each other and non-overlapping to createadjacent photon collection zones 210D and 211D, respectively, locatedbetween a near-range blanking region 212D and a far-range blankingregion 214D within the FOV 120 of the infrared camera 104. In a mannersimilar to that discussed above in conjunction with FIG. 2C, thelocation and width of each of the photon collection zones 210D and 211Dmay be set based on the width and delay of their corresponding exposurewindow 204D and 205D openings. In one example, each of the exposurewindow 204D and 205D openings may be 200 nsec, resulting in a depth of30 m for each of the photon collection zones 210D and 211D. In oneembodiment, the exposure window timing circuit 114 may generate theexposure windows 204D and 205D for the same infrared camera 104, whilein another example, the exposure window timing circuit 114 may generateeach of the exposure windows 204D and 205D for separate infrared cameras104. In the particular example of FIG. 2D, an object 101 located in thefirst photon collection zone 210D and not the second collection zone211D will result in returned light pulses 221D that are captured duringthe first exposure window 204D openings but not during the secondexposure window 205D openings.

In yet other embodiments, the exposure window timing circuit 114 maygenerate three or more multiple exposure windows 204D and 205D to yielda corresponding number of photon collection zones 210D and 211D, whichmay be located between the near-range blanking region 212D and thefar-range blanking region 214D. As indicated above, the multipleexposure windows 204D and 205D may correspond to infrared cameras 104.

In addition, while the exposure window 204D and 205D openings are of thesame length or duration as shown in FIG. 2D, other embodiments mayemploy varying durations for such exposure window 204D and 205Dopenings. In one example, each opening of the exposure window 204D and205D of increasing delay from its corresponding light pulse 202D mayalso be longer in duration, resulting in each photon collection regionor zone 210D and 211D being progressively greater in depth the moredistant that photon collection region 210D and 211D is from the infraredcamera 104. Associating the distance and the depth of each photoncollection region 210D and 211D in such a manner may help compensate fora loss of light reflected by an object 101, which is directlyproportional to the inverse of the square of the distance between thelight source 102 and the object 101. In some embodiments, the length ofeach photon collection region 210D and 211D may be proportional to, orotherwise related to, the inverse of the square of that distance.

FIG. 2E is a timing diagram of a recurring light pulse 202E generated bya light source 102 and multiple overlapping exposure window 204E and205E openings for the infrared camera 104 for fine resolution ranging ofobjects 101. In this example, the openings of the two exposure windows204E and 205E overlap in time. In the particular example of FIG. 2E, theopenings of the exposure windows 204E and 205E overlap by 50 percent,although other levels or degrees of overlap (e.g., 40 percent, 60percent, 80 percent, and so on) may be utilized in other embodiments.These exposure windows 204E and 205E thus result in overlapping photoncollection zones 210E and 211E, respectively, between a near-rangeblanking region 212E and a far-range blanking region 214E. Generally,the greater the amount of overlap of the exposure window 204E and 205Eopenings, the better the resulting depth resolution. For example,presuming a duration of each of the exposure windows 204E and 205E of500 nsec, resulting in each of the photon collection zones 210E and 211Ebeing 75 m deep, and presuming a 50 percent overlap of the openings ofexposure windows 204E and 205E, a corresponding overlap of the photoncollection zones 210E and 211E of 37.5 m is produced. Consequently, aneffective depth resolution of 37.5 meters may be achieved using exposurewindows 204E and 205E associated with photon collection zones 210E and211E of twice that depth.

For example, in the specific scenario depicted in FIG. 2E, returnedlight pulses 221E reflected by an object 101 are detected by theinfrared camera 104 using the first exposure window 204E, and are notdetected using the second exposure window 205E, indicating that theobject 101 is located within the first photon collection zone 210E butnot the second photon collection zone 211E. If, instead, the returnedlight pulses 221E reflected by the object 101 are detected using thesecond exposure window 205E, and are not detected using the firstexposure window 204E, the object 101 is located in the second photoncollection zone 211E and not the first photon collection zone 210E.Further, if the returned light pulses 221E reflected by the object 101are detected using the first exposure window 204E, and are also detectedusing the second exposure window 205E, the object 101 is located in theregion in which the first photon collection zone 210E and the secondphoton collection zone 211E overlap. Accordingly, presuming a 50 percentoverlap of the first exposure window 204E and the second exposure window205E, the location of the object 101 may be determined within a half ofthe depth of the first photon collection zone 210E and of the secondphoton collection zone 211E.

To implement the overlapped exposure windows, separate infrared cameras104 may be gated using separate ones of the first exposure window 204Eand the second exposure window 205E to allow detection of the two photoncollection zones 210E and 211E based on a single light pulse 202E. Inother examples in which a single infrared camera 104 is employed forboth the first exposure window 204E and the second exposure window 205E,the first exposure window 204E may be employed for light pulses 202E ofa photon collection cycle, and the second exposure window 205E may beused following other light pulses 202E of a separate photon collectioncycle. Thus, by tracking changes from one photon collection cycle toanother while dynamically altering the delay of the exposure window 204Eand 205E from the light pulses 202E, the location of objects 101detected within one of the photon collection zones 210E and 211E may bedetermined as described above.

While the examples of FIG. 2E involve the use of two exposure windows204E and 205E, the use of three or more exposure windows that overlapadjacent exposure windows may facilitate fine resolution detection ofthe distance from the infrared camera 104 to the object 101 over agreater range of depth.

While various alternatives presented above (e.g., the duration of thelight pulses 202, the duration of the openings of the exposure windows204 and their delay from the light pulses 202, the number of infraredcameras 104 employed, the collection of photons over a single ormultiple exposure window openings 204, and so on) are associated withparticular embodiments exemplified in FIGS. 2A through 2E, suchalternatives may be applied to other embodiments discussed inconjunction with FIGS. 2A through 2E, as well as to other embodimentsdescribed hereafter.

FIG. 3 is a timing diagram of light pulses 302A of one sensing system100A compared to an exposure window 304B for an infrared camera 104B ofa different sensing system 100B in which pulse timing diversity isemployed to mitigate intersystem interference. In one example, thesensing systems 100A and 100B may be located on separate vehicles, suchas two automobiles approaching one another along a two-way street.Further, each of the sensing systems 100A and 100B of FIG. 3 includes anassociated light source 102, an infrared camera 104, and a controlcircuit 110, as indicated in FIG. 1, but are not all explicitly showntherein to focus the following discussion. As depicted in FIG. 3, thelight pulses 302A of a light source 102A of the first sensing system100A, from time to time, may be captured during an exposure window 304Bgating the infrared camera 104B of the second sensing system 100B,possibly leading the range determination circuit 116 of the secondsensing system 100B to detect falsely the presence of an object 101 in aphoton collection zone corresponding to the exposure window 304B. Morespecifically, the first opening 320 of the exposure window 304B, asshown in FIG. 3, does not collect light from the first light pulse 302A,but the second occurrence of the light pulse 302A is shown arriving atthe infrared camera 104B during the second opening 322 of the exposurewindow 304B for the camera 104B. In one example, the range determinationcircuit 116 may determine that the number of photons collected at pixelsof the infrared camera 104B while the exposure window 304B is open istoo great to be caused by light reflected from an object 101, and maythus be received directly from a light source 102A that is notincorporated within the second sensing system 100B.

To address this potential interference, the sensing system 100B maydynamically alter the amount of time that elapses between at least twoconsecutive exposure window 304B openings (as well as betweenconsecutive light pulses generated by a light source of the secondsensing system 100B, not explicitly depicted in FIG. 3). For example, inthe case of FIG. 3, the third opening 324 of the exposure window 304Bhas been delayed, resulting in the third light pulse 302A being receivedat the infrared camera 104B of the second sensing system 100B prior tothe third opening 324 of the exposure window 304B. In this particularscenario, the exposure window timing circuit 114 of the second sensingsystem 100B has dynamically delayed the third opening 324 of theexposure window 304B in response to the range determination circuit 116detecting a number of photons being collected during the second opening322 of the exposure window 304B exceeding some threshold. Further, dueto the delay between the second opening 322 and the third opening 324 ofthe exposure window 304B, the fourth opening 326 of the exposure window304 is also delayed sufficiently to prevent collection of photons fromthe corresponding light pulse 302A from the light source 102A. In someexamples, the amount of delay may be predetermined, or may be morerandomized in nature.

In another embodiment, the exposure window timing circuit 114 of thesecond sensing system 100B may dynamically alter the timing betweenopenings of the exposure window 304B automatically, possibly in somerandomized manner. In addition, the exposure window timing circuit 114may make these timing alterations without regard as to whether the rangedetermination circuit 116 has detected collection of photons from thelight source 102A. In some examples, the light source timing circuit 112may alter the timing of the light pulses 302A from the light source102A, again, possibly in some randomized fashion. In yet otherimplementations, any combination of these measures (e.g., altered timingof the light pulses 302A and/or the exposure window 304B, randomlyand/or in response to photons captured directly instead of by reflectionfrom an object 101, etc.) may be employed.

Additional ways of mitigating intersystem interference other thanaltering the exposure window timing may also be utilized. FIG. 4 is agraph of multiple wavelength channels 402 for the light source 102 ofFIG. 1 to facilitate wavelength diversity. In one example, each sensingsystem 100 may be permanently assigned a particular wavelength channel402 at which the light source 102A may operate to generate light pulses.In the particular implementation of FIG. 4, ten separate wavelengthchannels 402 are available that span a contiguous wavelength range fromλ0 to λ10, although other numbers of wavelength channels 402 may beavailable in other examples. In other embodiments, the light sourcetiming circuit 112 may dynamically select one of the wavelength channels402. The selection may occur by way of activating one of severaldifferent light sources that constitute the light source 102 to providelight pulses at the selected wavelength channel 402. In other examples,the selection may occur by way of configuring a single light source 102to emit light at the selected wavelength channel 402. In a particularimplementation of FIG. 4, each wavelength channel 402 may possess a 5 nmbandwidth, with the ten channels collectively ranging from λ0=800 nm toλ10=850 nm. Other specific bandwidths, wavelengths, and number ofwavelength channels 402 may be employed in other examples, includingwavelength channels 402 that do not collectively span a contiguouswavelength range.

Correspondingly, the infrared camera 104 may be configured to detectlight in the wavelength channel 402 at which its corresponding lightsource 102 is emitting. To that end, the infrared camera 104 may beconfigured permanently to detect light within the same wavelengthchannel 402 at which the light source 102 operates. In another example,the exposure window timing circuit 114 may be configured to operate theinfrared camera 104 at the same wavelength channel 402 selected for thelight source 102. Such a selection, for example, may activate aparticular narrowband filter corresponding to the selected wavelengthchannel 402 so that light pulses at other wavelength channels 402 (e.g.,light pulses from other sensing systems 100) are rejected. Further, ifthe wavelength channel 402 to be used by the light source 102 and theinfrared camera 104 may be selected dynamically, such selections may bemade randomly over time and/or may be based on direct detection of lightpulses from other sensing systems 100, as discussed above in conjunctionwith FIG. 3.

FIG. 5A is a block diagram of an example multiple-camera sensing system500 to facilitate multiple FOVs 502, which may allow the sensing ofobjects 101 at greater overall angles than what may be possible with asingle infrared camera 104. In this example, nine different infraredcameras 104, each with its own FOV 502, are employed in a signalmulti-camera sensing system 500, which may be employed at a singlelocation, such as on a vehicle, thus providing nearly 360-degreecoverage of the area about the location. The infrared cameras 104 may beused in conjunction with the same number of light sources 102, or withgreater or fewer light sources. Further, the infrared cameras 104 mayemploy the same exposure window timing circuit 114, and may thus employthe same exposure window signals, or may be controlled by differentexposure window timing circuits 114 that may each provide differentexposure windows to each of the infrared cameras 114. In addition, theinfrared cameras 104 may detect light within the same range ofwavelengths, or light of different wavelength ranges. Other differencesmay distinguish the various infrared cameras of FIG. 5A as well.

Exhibiting how multiple infrared cameras 104 may be used in a differentway, FIG. 5B is a block diagram of an example multiple-camera sensingsystem 501 to facilitate multiple depth zones for approximately the sameFOV. In this particular example, a first infrared camera (e.g., infraredcamera 104A) is used to detect objects 101 within a near-range zone 512,a second infrared camera (e.g., infrared camera 104B) is used to detectobjects 101 within an intermediate-range zone 514, and a third infraredcamera (e.g., infrared camera 104C) is used to detect objects 101 withina far-range zone 516. Each of the infrared cameras 104 may be operatedusing any of the examples described above, such as those described inFIGS. 2A through 2E, FIG. 3, and FIG. 4. Additionally, while theparticular example of FIG. 5B defines the zones 512, 514, and 516 ofFIG. 5B as non-overlapping, multiple infrared cameras 104 may beemployed such that the zones 512, 514, and 516 corresponding to theinfrared camera 104A, 104B, and 104C, respectively, may at leastpartially overlap, as was described above with respect to theembodiments of FIGS. 2B and 2E.

FIG. 6 is a flow diagram of an example method 600 of using an infraredcamera for fine resolution ranging. While the method is described belowin conjunction with the infrared camera 104, the light source 102, thelight source timing circuit 112, and the range determination circuit 116of the sensing system 100 and variations disclosed above, otherembodiments of the method 600 may employ different devices or systemsnot specifically discussed herein.

In the method 600, the light source timing circuit 112 generates lightpulses using the light source 102 (operation 602). For each light pulse,the exposure window timing circuit 114 generates multiple exposurewindows for the infrared camera 104 (operation 604). Each of the windowscorresponds to a particular first range of distance from the infraredcamera 104. These windows may overlap in time in some examples. Therange determination circuit 116 may process the light captured at theinfrared camera 104 during the exposure windows to determine a secondrange of distance from the camera with a lower range uncertainty thanthe first range of distance (operation 606), as described in multipleexamples above.

While FIG. 6 depicts the operations 602-606 of the method 600 as beingperformed in a single particular order, the operations 602-606 may beperformed repetitively over some period of time to provide an ongoingindication of the distance of objects 101 from the infrared camera 104,thus potentially tracking the objects 101 as they move from one depthzone to another.

Consequently, in at least some embodiments of the sensing system 100 andthe method 600 described above, infrared cameras 104 may be employed notonly to determine the lateral or spatial location of objects relative tosome location, but to determine within some level of uncertainty thedistance of that location from the infrared cameras 104.

FIG. 7 is a block diagram of an example sensing system 700 including aninfrared camera 704 operating in conjunction with a controllable lightsource 702, and a light radar (lidar) system 706. By operating theinfrared camera 704 as described above with respect to the infraredcamera 104 of FIG. 1 according to one of the various embodimentsdiscussed earlier, and adding the use of the lidar system 706 in thesensing system 700, enhanced and efficient locating of objects 101 inthree dimensions may result.

The sensing system 700, as illustrated in FIG. 7, includes the lightsource 702, the infrared camera 704, the lidar system 706, and a controlcircuit 710. More specifically, the control circuit 710 includes aregion of interest (ROI) identifying circuit 712 and a range refiningcircuit 714. Each of the control circuit 710, the ROI identifyingcircuit 712, and the range refining circuit 714 may be implemented ashardware and/or software modules. The software modules may implementimage recognition algorithms and/or deep neural networks (DNN) that havebeen trained to detect and identify objects of interest. In someexamples, the ROI identifying circuit 712 may include the light sourcetiming circuit 112, the exposure window timing circuit 114, and therange determination circuit 116 of FIG. 1, such that the light source702 and the infrared camera 704 may be operated according to theembodiments discussed above to determine an ROI for each of the objects101 detected within a FOV 720 of the infrared camera 704. In at leastsome embodiments, an ROI may be a three-dimensional (or two-dimensional)region within which each of the objects 101 is detected. In someexamples further explained below, the ROI identifying circuit may employthe lidar system 706 in addition to the infrared camera 704 to determinethe ROI of each object 101. The range refining circuit 714 may thenutilize the lidar system 706 to probe each of the ROIs more specificallyto determine or refine the range of distance of the objects 101 from thesensing system 700.

In various embodiments of the sensing system 700, a “steerable” lidarsystem 706 that may be directed toward each of the identified ROIs isemployed to probe each ROI individually. FIG. 8A is a block diagram ofan example steerable lidar system 706A using a rotatable two-axis mirror810A. As shown, the lidar system 706A includes a sensor array 802A, azoom lens 804A, a narrow bandpass (NB) filter 806A, and possibly apolarizing filter 808A in addition to the two-axis mirror 810A. Inaddition, the steerable lidar system 706A may include its own lightsource (not shown in FIG. 8A), or may employ the light source 102employed by the infrared camera 104 of FIG. 1 to illuminate the ROI tobe probed. Other components may be included in the lidar system 706A, aswell as in the lidar system of FIG. 8B described hereafter, but suchcomponents are not discussed in detail hereinafter.

Alternatively, the lidar system 706 may include non-steerable lidar thatrepetitively and uniformly scans the scene at an effective frame ratethat may be less than that of the infrared camera 704. In this case, thelidar system 706 may provide high resolution depth measurements at ahigh spatial resolution for the selected ROIs while providing a morecoarse spatial sampling of points across the rest of the FOV. Byoperating the lidar system 706 in this way, the light source 702 and theinfrared camera 704 are primarily directed toward the ROIs. Thisalternative embodiment enables the use of uniform beam scanning hardware(e.g., polygon mirrors, resonant galvos, microelectromechanical systems(MEMS) mirrors) while reducing the overall light power and detectionprocessing requirements.

The sensor array 802A may be configured, in one example, as a square,rectangular array, or linear array of avalanche photodiodes (APDs) orsingle photon avalanche diodes (SPAD) 801 elements. The particularsensor array 802A of FIG. 8A is an 8×8 array, although other array sizesand shapes, as well as other element types, may be used in otherexamples. The zoom lens 804A may be operated or adjusted (e.g., by therange refining circuit 714) to control how many of the APDs 801 capturelight reflected from the object 101. For example, in a “zoomed in”position, the zoom lens 804 causes the object 101 to be detected usingmore of the APDs 801 than in a “zoomed out” position. In some examples,the zoom lens 804A may be configured or adjusted based on the size ofthe ROI being probed, with zoom lens 804A being configured to zoom infor smaller ROIs and to zoom out for larger ROIs. In other embodiments,the zoom lens 804A may be either zoomed in or out based on factors notrelated to the size of the ROI. Further, the zoom lens 804A may betelecentric, thus potentially providing the same magnification forobjects 101 at varying distances from the lidar system 706A.

The NB filter 806A may be employed in some embodiments to filter outlight at wavelengths that are not emitted from the particular lightsource being used to illuminate the object 101, thus reducing the amountof interference from other light sources that may disrupt adetermination of the distance of the object 101 from the lidar system706A. Also, the NB filter 806A may be switched out of the optical pathof the lidar system 706A, and/or additional NB filters 806A may beemployed so that the particular wavelengths being passed to the sensorarray 802A may be changed dynamically. Similarly, the polarizing filter808A may allow light of only a particular polarization that is optimizedfor the polarization of the light being used to illuminate the object101. If employed in the lidar system 706A, the polarizing filter 808Amay be switched dynamically out of the optical path of the lidar system706A if, for example, unpolarized light is being used to illuminate theobject 101.

The two-axis mirror 810A may be configured to rotate about both avertical axis and a horizontal axis to direct light reflected from anobject 101 in an identified ROI to the sensor array 802A via the filters808A and 806A and the zoom lens 804A. More specifically, the two-axismirror 810A may rotate about the vertical axis (as indicated by thedouble-headed arrow of FIG. 8A) to direct light from objects 101 atdifferent horizontal locations to the sensor array 802A, and may rotateabout the horizontal axis to direct light from objects 101 at differentvertical locations.

FIG. 8B is a block diagram of another example steerable or non-steerablelidar system 706B using a translatable zoom lens 804A instead of atwo-axis mirror. The steerable lidar system 706B also includes a sensorarray 802B employing multiple APDs 801 or other light detectionelements, as well as an NB filter 806B and possibly a polarizing filter808B. In the specific example of FIG. 8B, the zoom lens 804B translatesin a vertical direction to scan multiple horizontal swaths, one at atime, of the particular ROI being probed. To capture each swath, theparticular sensor array 802B may employ two offset rows of smaller,spaced-apart 8×8 arrays. Moreover, some of the columns of APDs 801between the upper row and the lower row of smaller arrays may overlap(as depicted in FIG. 8B), which may serve to reduce distortion of theresulting detected object 101 as the zoom lens 804B is translated up anddown by allowing the range refining circuit 714 or another controlcircuit to mesh together information from each scan associated with eachvertical position of the zoom lens 804B. However, while a particularconfiguration for the sensor array 802B is illustrated in FIG. 8B, manyother configurations for the sensor array 802B may be utilized in otherexamples.

The NB filter 806B and the polarizing filter 808B may be configured in amanner similar to the NB filter 806A and the polarizing filter 808A ofFIG. 8A described above. In one example, the NB filter 806B and thepolarizing filter 808B may be sized and/or shaped such that they mayremain stationary as the zoom lens 804B translates up and down in avertical direction.

Each lidar system 706A and 706B of FIGS. 8A and 8B may be a flash lidarsystem, in which a single light pulse from the lidar system 706A and706B is reflected from the object 101 back to all of the elements 801 ofthe sensor array 802A and 802B simultaneously. In such cases, the lidarsystem 706A and 706B may use the light source 102 (e.g., a VCSEL array)of the sensing system 100 of FIG. 1 to provide the light that is to bedetected at the sensor array 802A and 802B. In other examples, the lidarsystem 706 of FIG. 7 may instead be a scanning lidar system, in whichthe lidar system 706 provides its own light source (e.g., a laser) thatilluminates the object 101, with the reflected light being scanned overeach element of the sensor array 802 individually in succession, such asby way of a small, relatively fast rotating mirror.

FIG. 9 is a flow diagram of an example method 900 of employing aninfrared camera (e.g., the infrared camera 704 of FIG. 7) and a lidarsystem (e.g., the lidar system 706 of FIG. 7) for fine range resolution.In the method 900, an ROI and a first range of distance to the ROI isidentified using the infrared camera (operation 902). This may beaccomplished using image recognition algorithms or DNN that have beentrained to detect and identify the objects of interest. The ROI is thenprobed using the lidar system to refine the first range to a secondrange of distance to the ROI having a lower measurement uncertainty(operation 904). Consequently in at least some embodiments, the sensingsystem 700 of FIG. 7 may employ the infrared camera 704 and thesteerable lidar system 706 in combination to provide significantresolution regarding the location of objects both radially (e.g., in a zdirection) and spatially, or laterally and vertically (e.g., in an x, yplane orthogonal to the z direction), beyond the individual capabilitiesof either the infrared camera 704 or the lidar system 706. Morespecifically, the infrared camera 704, which typically may facilitatehigh spatial resolution but less distance or depth resolution, is usedto generate an ROI for each detected object in a particular scene. Thesteerable lidar system 706, which typically provides superior distanceor radial resolution but less spatial resolution, may then probe each ofthese ROIs individually, as opposed to probing the entire scene indetail, to more accurately determine the distance to the object in theROI with less range uncertainty.

Moreover, the inclusion of additional sensors or equipment in a systemthat utilizes an infrared camera and a steerable lidar system mayfurther enhance the object sensing capabilities of the system. FIG. 10is a block diagram of an example vehicle autonomy system 1000 in whichnear-infrared (NIR) range-gated cameras 1002 and steerable lidar systems1022, in conjunction with other sensors and components may be employedto facilitate navigational control of a vehicle, such as, for example,an electrically-powered automobile.

As depicted in FIG. 10, the vehicle autonomy system 1000, in addition toNIR range-gated cameras 1002 and steerable lidar systems 1022, mayinclude a high dynamic range (HDR) color camera 1006, a camerapreprocessor 1004, VCSEL clusters 1010, a VCSEL pulse controller 1008, alidar controller 1020, one or more additional sensors 1016, a long-waveinfrared (LWIR) microbolometer camera 1014, a biological detectionpreprocessor 1012, a vehicle autonomy processor 1030, and vehiclecontrollers 1040. Other components or devices may be incorporated in thevehicle autonomy system 1000, but are not discussed herein to simplifyand focus the following discussion.

The VCSEL clusters 1010 may be positioned at various locations about thevehicle to illuminate the surrounding area with NIR light for use by theNIR range-gated cameras 1002, and possibly by the steerable lidarsystems 1022, to detect objects (e.g., other vehicles, pedestrians, roadand lane boundaries, road obstacles and hazards, warning signs, trafficsignals, and so on). In one example, each VCSEL cluster 1010 may includeseveral lasers providing light at wavelengths in the 800 to 900 nm rangeat a total cluster laser power of 2-4 W. Each cluster may be spaced atleast 250 mm in some embodiments to meet reduced accessible emissionlevels. However, other types of light sources with differentspecifications may be employed in other embodiments. In at least someexamples, the VCSEL clusters 1010 may serve as a light source (e.g., thelight source 102 of FIG. 1), as explained above.

The VCSEL cluster pulse controller 1008 may be configured to receivepulse mode control commands and related information from the vehicleautonomy processor 1030 and drive or pulse the VCSEL clusters 1010accordingly. In at least some embodiments, the VCSEL cluster pulsecontroller 1008 may serve as a light source timing circuit (e.g., thelight source timing circuit 112 of FIG. 1), thus providing the variouslight pulsing modes for illumination, range-gating of the NIRrange-gated cameras 1002, and the like, as discussed above.

The NIR range-gated cameras 1002 may be configured to identify ROIsusing the various range-gating techniques facilitated by the opening andclosing of the camera exposure window, thus potentially serving as aninfrared camera (e.g., the infrared camera 104 of FIG. 1), as discussedearlier. In some examples, the NIR range-gated cameras 1002 may bepositioned about the vehicle to facilitate FOV coverage about at least amajority of the environment of the vehicle. In one example, each NIRrange-gated camera 1002 may be a high dynamic range (HDR) NIR cameraincluding an array (e.g., a 2K×2K array) of imaging elements, asmentioned earlier, although other types of infrared cameras may beemployed in the vehicle autonomy system 1000.

The camera preprocessor 1004 may be configured to open and close theexposure windows of each of the NIR range-gated cameras 1002, and thusmay serve in some examples as an exposure window timing circuit (e.g.,the exposure window timing circuit 114 of FIG. 1), as discussed above.In other examples, the camera preprocessor 1004 may receive commandsfrom the vehicle autonomy processor 1030 indicating the desired exposurewindow timing, which may then operate the exposure windows accordingly.The camera preprocessor 1004 may also read or receive the resultingimage element data (e.g., pixel voltages resulting from exposure of theimage elements to light) and processing that data to determine the ROIs,including their approximate distance from the vehicle, associated witheach object detected based on differences in light received at eachimage element, in a manner similar to that of the ROI identificationcircuit 712 of FIG. 7. The determination of an ROI may involve comparingthe image data of the elements to some threshold level for a particulardepth to determine whether an object has been detected within aparticular collection zone, as discussed above. In some embodiments, thecamera preprocessor 1004 may perform other image-related functions,possibly including, but not limited to, image segmentation (in whichmultiple objects, or multiple features of a single object, may beidentified) and image fusion (in which information regarding an objectdetected in multiple images may be combined to yield more specificinformation describing that object).

In some examples, the camera preprocessor 1004 may also becommunicatively coupled with the HDR color camera 1006 (or multiple suchcameras) located on the vehicle. The HDR color camera 1006 may include asensor array capable of detecting varying colors of light to distinguishvarious light sources in an overall scene, such as the color of trafficsignals or signs within view. During low-light conditions, such as atnight, dawn, and dusk, the exposure time of the HDR color camera 1006may be reduced to prevent oversaturation or “blooming” of the sensorarray imaging elements to more accurately identify the colors of brightlight sources. Such a reduction in exposure time may be possible in atleast some examples since the more accurate determination of thelocation of objects is within the purview of the NIR range-gated cameras1002 and the steerable lidar systems 1022.

The camera preprocessor 1004 may also be configured to control theoperation of the HDR color camera 1006, such as controlling the exposureof the sensor array imaging elements, as described above, possibly underthe control of the vehicle autonomy processor 1030. In addition, thecamera preprocessor 1004 may receive and process the resulting imagedata from the HDR color camera 1006 and forward the resulting processedimage data to the vehicle autonomy processor 1030.

In some embodiments, the camera preprocessor 1004 may be configured tocombine the processed image data from both the HDR color camera 1006 andthe NIR range-gated cameras 1002, such as by way of image fusion and/orother techniques, to relate the various object ROIs detected using theNIR range-gated cameras 1002 with any particular colors detected at theHDR color camera 1006. Moreover, camera preprocessor 1004 may storeconsecutive images of the scene or environment surrounding the vehicleand perform scene differencing between those images to determine changesin location, color, and other aspects of the various objects beingsensed or detected. As is discussed more fully below, the use of suchinformation may help the vehicle autonomy system 1000 determine whetherits current understanding of the various objects being detected remainsvalid, and if so, may reduce the overall data transmission bandwidth andsensor data processing that is to be performed by the vehicle autonomyprocessor 1030.

Each of the steerable lidar systems 1022 may be configured as a lidarsystem employing a two-axis mirror (e.g., the lidar system 706A of FIG.8A), a lidar system employing a translatable lens (e.g., the lidarsystem 706B of FIG. 8B), or another type of steerable lidar system notspecifically described herein. Similar to the lidar system 706 of FIG.7, the steerable lidar systems 1022 may probe the ROIs identified by theNIR range-gated cameras 1002 and other components of the vehicleautonomy system 1000 under the control of the lidar controller 1020. Insome examples, the lidar controller 1020 may provide functionalitysimilar to the range refining circuit 714 of FIG. 7, as described above.Further, the lidar controller 1020, possibly in conjunction with thecamera preprocessor 1004 and/or the vehicle autonomy processor 1030, mayperform scene differencing using multiple scans, as described above, totrack objects as they move through the scene or area around the vehicle.Alternatively, the lidar system may be the non-steerable lidar systemdescribed above that provides selective laser pulsing and detectionprocessing.

The LWIR microbolometer camera 1014 may be a thermal (e.g., infrared)camera having a sensor array configured to detect, at each of itsimaging elements, thermal radiation typically associated with humans andvarious animals. The biological detection preprocessor 1012 may beconfigured to control the operation of the LWIR microbolometer camera1014, possibly in response to commands received from the vehicleautonomy processor 1030. Additionally, the biological detectionpreprocessor 1012 may process the image data received from the LWIRmicrobolometer camera 1014 to help identify whether any particularimaged objects in the scene are human or animal in nature, as well aspossibly to specifically distinguish humans from other thermal sources,such as by way of intensity, size, and/or other characteristics.

Other sensors 1016 not specifically mentioned above may also be includedin the vehicle autonomy system 1000. Such sensors 1016 may include, butare not limited to, radar systems and other sensors for additionalobject sensing or detection, as well as inertial measurement units(IMUs), which may provide acceleration, velocity, orientation, and othercharacteristics regarding the current position and movement of thevehicle. The other sensors 1016 may be controlled by the vehicleautonomy processor 1030 or another processor not explicitly indicated inFIG. 10, and the resulting sensor data may be provided to the vehicleautonomy processor 1030 or another processor for analysis in view of thedata received from the NIR range-gated cameras 1002, the steerable lidarsystems 1022, and other components of the vehicle autonomy system 1000.The other sensors 1016 may also include human input sensors, such assteering, acceleration, and braking input that may be provided by anoccupant of the vehicle.

The vehicle autonomy processor 1030 may communicate directly orindirectly with the various cameras, sensors, controllers, andpreprocessors, as discussed above, to determine the location, andpossibly the direction and speed of movement, of the objects detected inthe area around the vehicle. Based on this information, as well as onnavigational information, speed limit data, and possibly otherinformation, the vehicle autonomy processor 1030 may control the vehiclevia the vehicle controllers 1040 to operate the motor, brakes, steeringapparatus, and other aspects of the vehicle. The vehicle controllers1040 may include, but are not limited to, an acceleration controller, abraking controller, a steering controller, and so on. Such control bythe vehicle autonomy processor 1030 may be fully autonomous orsemiautonomous (based at least partially on, for example, the humansteering, acceleration, and braking input mentioned above).

The vehicle autonomy processor 1030, the camera preprocessor 1004, thelidar controller 1020, the VCSEL pulse controller 1008, the biologicaldetection preprocessor 1012, or the vehicle controllers 1040 may includeanalog and/or digital electronic circuitry, and/or may includemicrocontrollers, DSPs, and/or other algorithmic processors configuredto execute software or firmware instructions stored in a memory toperform the various functions ascribed to each of these components.

FIG. 11 is a flow diagram of an example method 1100 of operating avehicle autonomy system, such as the vehicle autonomy system 1000 ofFIG. 10. In at least some embodiments, the various operations of themethod 1100 are executed and/or controlled by the vehicle autonomyprocessor 1030 working in conjunction with the various preprocessors andcontrollers, such as the camera preprocessor 1004, the lidar controller1020, the VCSEL pulse controller 1008, and the biological detectionpreprocessor 1012. In the method 1100, input from sensors (e.g., the NIRrange-gated cameras 1002 operating in range-gated mode, the HDR colorcamera 1006, the LWIR microbolometer camera 1014, and/or other sensors1016, such as radars, IMUs, and the like) may be processed to identifyROIs in which o objects may be located (operation 1102). In someexamples, input from the steerable lidar systems 1022 operating in a“raster scanning” mode (e.g., scanning over the entire viewable scene orarea, as opposed to focusing on a particular ROI) may also be used. Inone embodiment, the camera image data and sensor data may be combined(e.g., by image fusion in some examples) to correlate detected imagesand their various aspects (e.g., area size, color, and/or so forth) toidentify the ROIs.

The steerable lidar systems 1022 may then be operated to probe each ofthe identified ROIs (operation 1104), such as to more accuratelydetermine a depth or distance of each corresponding object from thevehicle. To control the steerable lidar systems 1022 to perform theprobing function, information describing each identified ROI, including,for example, spatial location, approximate distance, and size and/orshape data, may be processed to yield control information useful inoperating the steerable lidar systems 1022 in probing each ROI. Thiscontrol information may include, for example, lidar steering coordinatesfor each ROI, spatial sample size (e.g., width and height) for each ROI(useful for setting a zoom level for the lidar systems 1022 in at leastsome cases), scanning pattern for each ROI, and/or laser pulserepetition rates for the VCSEL clusters 1010 or dedicated light sourcesfor the lidar systems 1022 so that the lidar systems 1022 may probe eachROI to yield the more specific distance information. In someembodiments, this information may be in the form of a range map andassociated amplitudes of the light being reflected or returned.

Once such detailed location and other information has been obtainedregarding each object, the vehicle autonomy processor 1030 and/or thelidar controller 1020 may continue to operate the steerable lidarsystems 1022 to probe the various ROIs in conjunction with informationthat continues to be received from any or all of the NIR range-gatedcameras 1002, the HDR color camera 1006, the LWIR microbolometer camera1014, and the other sensors 1016. Using this input, the vehicle autonomyprocessor 1030, the camera preprocessor 1004, and/or the lidarcontroller 1020 track scene-to-scene differences. If the scene remainsunderstandable and/or coherent to the vehicle autonomy system 1000and/or other components of the vehicle autonomy system 1000 (operation1108), the lidar controller 1020 may continue to operate the steerablelidar systems 1022 to probe each ROI (operation 1106). In such cases,the boundaries of the ROI may change over time as the object beingtracked moves relative to the vehicle. Operating in this mode, in atleast some examples, possibly alleviates the vehicle autonomy processor1030, as well as the camera preprocessor 1004 and other components ofthe vehicle autonomy system 1000, from the processing associated withreacquiring each object and determining its associated ROI.

If, instead, the vehicle autonomy processor 1030 or another processor(e.g., the lidar controller 1020, the camera preprocessor 1004, and/orthe biological detection preprocessor 1012) loses understanding of thescene (operation 1108), the vehicle autonomy processor 1030 may returnthe system back to an ROI identification mode (operation 1102),employing the NIR range-gated cameras, the steerable lidar systems 1022in raster scanning mode, the HDR color camera 1006, the LWIRmicrobolometer camera 1014, and/or the other sensors 1016 to identifythe current ROIs to be probed using the steerable lidar systems 1022(operation 1104). In at least some examples, the vehicle autonomy system1000 may lose understanding of the current scene in ways, such as, forexample, losing track of an object that was recently located in thescene, an unexpected appearance of an object within the scene withoutbeing detected previously, an unexpected movement or change of directionof an object being tracked, other temporal inconsistencies and/ordiscrepancies between object positions and/or identities, and so on.

Based on the sensing of the objects in the area surrounding the vehicle,the vehicle autonomy processor 1030 may issue commands to the vehiclecontrollers 1040 to navigate the vehicle to avoid the detected objects(e.g., obstacles or hazards that may pose a risk), operate the vehicleaccording to detected warning signs and traffic signals, and so on.

Turning to FIG. 12, an electronic device 1200 including operationalunits 1202-1212 arranged to perform various operations of the presentlydisclosed technology is shown. The operational units 1202-1212 of thedevice 1200 may be implemented by hardware or a combination of hardwareand software to carry out the principles of the present disclosure. Itwill be understood by persons of skill in the art that the operationalunits 1202-1212 described in FIG. 12 may be combined or separated intosub-blocks to implement the principles of the present disclosure.Therefore, the description herein supports any possible combination orseparation or further definition of the operational units 1202-1212.Moreover, multiple electronic devices 1200 may be employed in variousembodiments.

In one implementation, the electronic device 1200 includes an outputunit 1202 configured to provide information, including possibly displayinformation, such as by way of a graphical user interface, and aprocessing unit 1204 in communication with the output unit 1202 and aninput unit 1206 configured to receive data from input devices orsystems. Various operations described herein may be implemented by theprocessing unit 1204 using data received by the input unit 1206 tooutput information using the output unit 1202.

Additionally, in one implementation, the electronic device 1200 includescontrol units 1208 implementing the operations 602-606, 902-904, and1102-1108 of FIGS. 6, 9, and 11. Accordingly, the control units 1208 mayinclude or perform the operations associated with the control circuit110 of FIG. 1, including the light source timing circuit 112, theexposure window timing circuit 114, and/or the range determinationcircuit 116, as well as the control circuit 710 of FIG. 7, including theROI identification circuit 712 and/or the range refining circuit 714.Further, the electronic device 1200 may serve as any of the controllersand/or processors of FIG. 10, such as the camera preprocessor 1004, theVCSEL pulse controller 1008, the biological detection preprocessor 1012,the lidar controller 1020, the vehicle autonomy processor 1030, and/orthe vehicle controllers 1040.

Referring to FIG. 13, a detailed description of an example computingsystem 1300 having computing units that may implement various systemsand methods discussed herein is provided. The computing system 1300 maybe applicable to, for example, the sensing systems 100, 500, 501, and/or700, and similar systems described herein, as well as the vehicleautonomy system 1000, and various control circuits, controllers,processors, and the like described in connection thereto. It will beappreciated that specific implementations of these devices may be ofdiffering possible specific computing architectures not all of which arespecifically discussed herein but will be understood by those ofordinary skill in the art.

The computer system 1300 may be a computing system capable of executinga computer program product to execute a computer process. Data andprogram files may be input to the computer system 1300, which reads thefiles and executes the programs therein. Some of the elements of thecomputer system 1300 are shown in FIG. 13, including hardware processors1302, data storage devices 1304, memory devices 1306, and/or ports1308-1312. Additionally, other elements that will be recognized by thoseskilled in the art may be included in the computing system 1300 but arenot explicitly depicted in FIG. 13 or discussed further herein. Variouselements of the computer system 1300 may communicate with one another byway of communication buses, point-to-point communication paths, or othercommunication means not explicitly depicted in FIG. 13.

The processor 1302 may include, for example, a central processing unit(CPU), a microprocessor, a microcontroller, a digital signal processor(DSP), and/or internal levels of cache. There may be processors 1302,such that the processor 1302 comprises a single central-processing unit,or processing units capable of executing instructions and performingoperations in parallel with each other, commonly referred to as aparallel processing environment.

The computer system 1300 may be a conventional computer, a distributedcomputer, or any other type of computer, such as external computers madeavailable via a cloud computing architecture. The presently describedtechnology is optionally implemented in software stored on the datastorage device(s) 1304, stored on the memory device(s) 1306, and/orcommunicated via the ports 1308-1312, thereby transforming the computersystem 1300 in FIG. 13 to a special purpose machine for implementing theoperations described herein. Examples of the computer system 1300include personal computers, terminals, workstations, mobile phones,tablets, laptops, personal computers, multimedia consoles, gamingconsoles, set top boxes, embedded computing and processing systems, andthe like.

The data storage devices 1304 may include any non-volatile data storagedevice capable of storing data generated or employed within thecomputing system 1300, such as computer executable instructions forperforming a computer process, which may include instructions of bothapplication programs and an operating system (OS) that manages thevarious components of the computing system 1300. The data storagedevices 1304 may include, without limitation, magnetic disk drives,optical disk drives, solid state drives (SSDs), flash drives, and thelike. The data storage devices 1304 may include removable data storagemedia, non-removable data storage media, and/or external storage devicesmade available via a wired or wireless network architecture with suchcomputer program products, including database management products, webserver products, application server products, and/or other additionalsoftware components. Examples of removable data storage media includeCompact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc Read-OnlyMemory (DVD-ROM), magneto-optical disks, flash drives, and the like.Examples of non-removable data storage media include internal magnetichard disks, SSDs, and the like. The memory devices 1306 may includevolatile memory (e.g., dynamic random access memory (DRAM), staticrandom access memory (SRAM), etc.) and/or non-volatile memory (e.g.,read-only memory (ROM), flash memory, etc.).

Computer program products containing mechanisms to effectuate thesystems and methods in accordance with the presently describedtechnology may reside in the data storage devices 1304 and/or the memorydevices 1306, which may be referred to as machine-readable media. Itwill be appreciated that machine-readable media may include any tangiblenon-transitory medium that is capable of storing or encodinginstructions to perform any of the operations of the present disclosurefor execution by a machine or that is capable of storing or encodingdata structures and/or modules utilized by or associated with suchinstructions. Machine-readable media may include a single medium ormultiple media (e.g., a centralized or distributed database, and/orassociated caches and servers) that store the executable instructions ordata structures.

In some implementations, the computer system 1300 includes ports, suchas an input/output (I/O) port 1308, a communication port 1310, and asub-systems port 1312, for communicating with other computing, network,or vehicle devices. It will be appreciated that the ports 1308-1312 maybe combined or separate and that more or fewer ports may be included inthe computer system 1300.

The I/O port 1308 may be connected to an I/O device, or other device, bywhich information is input to or output from the computing system 1300.Such I/O devices may include, without limitation, input devices, outputdevices, and/or environment transducer devices.

In one implementation, the input devices convert a human-generatedsignal, such as, human voice, physical movement, physical touch orpressure, and/or the like, into electrical signals as input data intothe computing system 1300 via the I/O port 1308. Similarly, the outputdevices may convert electrical signals received from computing system1300 via the I/O port 1308 into signals that may be sensed as output bya human, such as sound, light, and/or touch. The input device may be analphanumeric input device, including alphanumeric and other keys forcommunicating information and/or command selections to the processor1302 via the I/O port 1308. The input device may be another type of userinput device including, but not limited to: direction and selectioncontrol devices, such as a mouse, a trackball, cursor direction keys, ajoystick, and/or a wheel; sensors, such as a camera, a microphone, apositional sensor, an orientation sensor, a gravitational sensor, aninertial sensor, and/or an accelerometer; and/or a touch-sensitivedisplay screen (“touchscreen”). The output devices may include, withoutlimitation, a display, a touchscreen, a speaker, a tactile and/or hapticoutput device, and/or the like. In some implementations, the inputdevice and the output device may be the same device, for example, in thecase of a touchscreen.

The environment transducer devices convert one form of energy or signalinto another for input into or output from the computing system 1300 viathe I/O port 1308. For example, an electrical signal generated withinthe computing system 1300 may be converted to another type of signal,and/or vice-versa. In one implementation, the environment transducerdevices sense characteristics or aspects of an environment local to orremote from the computing device 1300, such as, light, sound,temperature, pressure, magnetic field, electric field, chemicalproperties, physical movement, orientation, acceleration, gravity,and/or the like. Further, the environment transducer devices maygenerate signals to impose some effect on the environment either localto or remote from the example computing device 1300, such as, physicalmovement of some object (e.g., a mechanical actuator), heating orcooling of a substance, adding a chemical substance, and/or the like.

In one implementation, a communication port 1310 is connected to anetwork by way of which the computer system 1300 may receive networkdata useful in executing the methods and systems set out herein as wellas transmitting information and network configuration changes determinedthereby. Stated differently, the communication port 1310 connects thecomputer system 1300 to communication interface devices configured totransmit and/or receive information between the computing system 1300and other devices by way of wired or wireless communication networks orconnections. Examples of such networks or connections include, withoutlimitation, Universal Serial Bus (USB), Ethernet, Wi-Fi, Bluetooth®,Near Field Communication (NFC), Long-Term Evolution (LTE), and so on.Such communication interface devices may be utilized via thecommunication port 1310 to communicate other machines, either directlyover a point-to-point communication path, over a wide area network (WAN)(e.g., the Internet), over a local area network (LAN), over a cellular(e.g., third generation (3G) or fourth generation (4G)) network, or overanother communication means. Further, the communication port 1310 maycommunicate with an antenna for electromagnetic signal transmissionand/or reception. In some examples, an antenna may be employed toreceive Global Positioning System (GPS) data to facilitate determinationof a location of a machine, vehicle, or another device.

The computer system 1300 may include a sub-systems port 1312 forcommunicating with systems related to a vehicle to control an operationof the vehicle and/or exchange information between the computer system1300 and sub-systems of the vehicle. Examples of such sub-systems of avehicle, include, without limitation, imaging systems, radar, lidar,motor controllers and systems, battery control, fuel cell or otherenergy storage systems or controls in the case of such vehicles withhybrid or electric motor systems, autonomous or semi-autonomousprocessors and controllers, steering systems, brake systems, lightsystems, navigation systems, environment controls, entertainmentsystems, and the like.

In an example implementation, object sensing information and softwareand other modules and services may be embodied by instructions stored onthe data storage devices 1304 and/or the memory devices 1306 andexecuted by the processor 1302. The computer system 1300 may beintegrated with or otherwise form part of a vehicle. In some instances,the computer system 1300 is a portable device that may be incommunication and working in conjunction with various systems orsub-systems of a vehicle.

The present disclosure recognizes that the use of such information maybe used to the benefit of users. For example, the sensing information ofa vehicle may be employed to provide directional, acceleration, braking,and/or navigation information, as discussed above. Accordingly, use ofsuch information enables calculated control of an autonomous vehicle.Further, other uses for location information that benefit a user of thevehicle are also contemplated by the present disclosure.

Users can selectively block use of, or access to, personal data, such aslocation information. A system incorporating some or all of thetechnologies described herein can include hardware and/or software thatprevents or blocks access to such personal data. For example, the systemcan allow users to “opt in” or “opt out” of participation in thecollection of personal data or portions thereof. Also, users can selectnot to provide location information, or permit provision of generallocation information (e.g., a geographic region or zone), but notprecise location information.

Entities responsible for the collection, analysis, disclosure, transfer,storage, or other use of such personal data should comply withestablished privacy policies and/or practices. Such entities shouldsafeguard and secure access to such personal data and ensure that otherswith access to the personal data also comply. Such entities shouldimplement privacy policies and practices that meet or exceed industry orgovernmental requirements for maintaining the privacy and security ofpersonal data. For example, an entity should collect users' personaldata for legitimate and reasonable uses and not share or sell the dataoutside of those legitimate uses. Such collection should occur onlyafter receiving the users' informed consent. Furthermore, third partiescan evaluate these entities to certify their adherence to establishedprivacy policies and practices.

The system set forth in FIG. 13 is but one possible example of acomputer system that may employ or be configured in accordance withaspects of the present disclosure. It will be appreciated that othernon-transitory tangible computer-readable storage media storingcomputer-executable instructions for implementing the presentlydisclosed technology on a computing system may be utilized.

In the present disclosure, the methods disclosed may be implemented assets of instructions or software readable by a device. Further, it isunderstood that the specific order or hierarchy of steps in the methodsdisclosed are instances of example approaches. Based upon designpreferences, it is understood that the specific order or hierarchy ofsteps in the method can be rearranged while remaining within thedisclosed subject matter. The accompanying method claims presentelements of the various steps in a sample order, and are not necessarilymeant to be limited to the specific order or hierarchy presented.

The described disclosure may be provided as a computer program product,or software, that may include a non-transitory machine-readable mediumhaving stored thereon instructions, which may be used to program acomputer system (or other electronic devices) to perform a processaccording to the present disclosure. A machine-readable medium includesany mechanism for storing information in a form (e.g., software,processing application) readable by a machine (e.g., a computer). Themachine-readable medium may include, but is not limited to, magneticstorage medium, optical storage medium; magneto-optical storage medium,read only memory (ROM); random access memory (RAM); erasableprogrammable memory (e.g., EPROM and EEPROM); flash memory; or othertypes of medium suitable for storing electronic instructions.

While the present disclosure has been described with reference tovarious implementations, it will be understood that theseimplementations are illustrative and that the scope of the disclosure isnot so limited. Many variations, modifications, additions, andimprovements are possible. More generally, implementations in accordancewith the present disclosure have been described in the context ofparticular implementations. Functionality may be separated or combinedin blocks differently in various embodiments of the disclosure ordescribed with different terminology. These and other variations,modifications, additions, and improvements may fall within the scope ofthe disclosure as defined in the claims that follow.

What is claimed is:
 1. A system for sensing objects, the systemcomprising: a light source; a camera; a light radar (lidar) system; aregion of interest identification circuit identifying a region ofinterest corresponding to an object and a first range of distance to theregion of interest, the first range of distance to the region ofinterest being determined based on a delayed timing relationship betweenthe camera and the light source, the delayed timing relationshipincluding an illumination of the light source followed by a delay for aperiod of time corresponding to the first range of distance and anopening of an exposure window of the camera at a first time after theperiod of time has elapsed, the exposure window being closed at a secondtime, the first time corresponding to a first edge of a photoncollection zone and the second time corresponding to a second edge ofthe photon collection zone, the region of interest being in the photoncollection zone; and a range refining circuit refining the first rangeto a second range of distance to the region of interest by probing theregion of interest using the lidar system, the second range having alower uncertainty than the first range.
 2. The system of claim 1,wherein the region of interest identification circuit identifies theregion of interest and the first range of distance to the region ofinterest using the lidar system in addition to the camera and the lightsource.
 3. The system of claim 1, wherein the light source comprises avertical-cavity surface-emitting laser (VCSEL) array.
 4. The system ofclaim 1, wherein: the light source comprises a near-infrared (NIR) lightsource; and the camera comprises a NIR camera.
 5. The system of claim 1,wherein: the lidar system comprises a flash lidar system; and the rangerefining circuit probes the region of interest using the light source inconjunction with the lidar system.
 6. The system of claim 1, wherein:the light source comprises a first light source; and the system furthercomprises a second light source different than the first light source,wherein the lidar system comprises a scanning lidar system, and therange refining circuit probes the region of interest using the secondlight source in conjunction with the lidar system.
 7. A method forsensing objects, the method comprising: illuminating a light source;delaying an opening of an exposure window of a camera for a period oftime following the light source being illuminated according to a delayedtiming relationship corresponding to a first range of distance from thecamera, the period of time being associated with a photon collectionzone within field of view of the camera; opening the exposure window ata first time after the period of time has elapsed according to thedelayed timing relationship, the exposure window being closed at asecond time, the first time corresponding to a first edge of the photoncollection zone and the second time corresponding to a second edge ofthe photon collection zone; identifying a region of interestcorresponding to an object located within the first range of distancefrom the camera based on the delayed timing relationship, the region ofinterest being in the photon collection zone; and refining the firstrange of distance to a second range of distance to the region ofinterest using a lidar system, the second range of distance having alower uncertainty than the first range of distance.
 8. The method ofclaim 7, further comprising identifying the region of interest and thefirst range of distance to the region of interest using the lidar systemin addition to the camera and the light source.
 9. The method of claim7, wherein the light source comprises a vertical-cavity surface-emittinglaser (VCSEL) array.
 10. The method of claim 7, wherein: the lightsource comprises a near-infrared (NIR) light source; and the cameracomprises a NIR camera.
 11. The method of claim 7, wherein: the lidarsystem comprises a flash lidar system; and the method further comprisingprobing the region of interest using the light source in conjunctionwith the lidar system.
 12. The method of claim 7, wherein: the lightsource comprises a first light source and a second light sourcedifferent from the first light source; and wherein the lidar systemcomprises a scanning lidar system, and the method further comprisingprobing the region of interest using the second light source inconjunction with the lidar system.
 13. The method of claim 7, whereinthe first edge is adjacent a near-range blanking region and the secondedge is adjacent a far-range blanking region.
 14. A non-transitorycomputer-readable medium storing instructions that, when executed by aprocessor, cause the processor to perform operations for sensingobjects, the operations comprising: generating a delayed timingrelationship between a light source and a camera, the delayed timingrelationship including an illumination of the light source followed by adelay for a period of time, an opening of an exposure window of thecamera at a first time after the period of time has elapsed, and aclosing of the exposure window of the camera at a second time, theperiod of time being associated with a photon collection zone withinfield of view of the camera, the first time corresponding to a firstedge of the photon collection zone and the second time corresponding toa second edge of the photon collection zone; identifying a region ofinterest corresponding to an object and a first range of distance to theregion of interest, the region of interest being in the photoncollection zone, the first range of distance to the region of interestbeing identified based on the delayed timing relationship between thecamera and the light source; and refining the first range of distance toa second range of distance to the region of interest, the second rangeof distance having a lower uncertainty than the first range of distance.15. The non-transitory computer-readable medium of claim 14, wherein thefirst range of distance is refined to the second range of distance basedon a lidar system probing the region of interest.
 16. The non-transitorycomputer-readable medium of claim 14, wherein the light source comprisesa vertical-cavity surface-emitting laser (VCSEL) array.
 17. Thenon-transitory computer-readable medium of claim 14, wherein: the lightsource comprises a near-infrared (NIR) light source; and the cameracomprises a NIR camera.